MAGAZINE 


FOR NOVICE AND ADVANCED USERS 


FreeBSD Tools 


DRAGONFLYBSD PACKAGE MANAGEMENT SYSTEM 


DATABASE QUERY 
OPTIMIZATION 
BRETT DAVIS 
FOR HUGE DATABASES OF IXSYSTEMS 
TELLS YOU ALL 
ABOUT TRUENA 
IS THERE A DOWN SIDE y oo 


TO MASS CONNECTIVITY? ft. 


HOW IMPORTANT IS YOUR DATA? 


Years of family photos. Your entire music 
and movie collection. Office documents 
you've put hours of work into. Backups for 
every computer you own. We ask again, how 
important is your data? 


NOW IMAGINE LOSING IT ALL 


Losing one bit - that’s all it takes. One single bit, and 
your file is gone. 


The worst part? You won't know until you 
absolutely need that file again. 


THE SOLUTION 


The FreeNAS Mini has emerged as the clear choice to 
save your digital life. No other NAS in its class offers 
ECC (error correcting code) memory and ZFS bitrot 
protection to ensure data always reaches disk 
without corruption and never degrades over time. 


No other NAS combines the inherent data integrity 
and security of the ZFS filesystem with fast on-disk 
encryption. No other NAS provides comparable power 
and flexibility. The FreeNAS Mini is, hands-down, the 
best home and small office storage appliance you can 
buy on the market. When it comes to saving your 
important data, there simply is no other solution. 


Example of one-bit corruption 


The Mini boasts these state-of-the- 
art features: 


- 8-core 2.4GHz Intel® Atom™ processor 

« Up to 16TB of storage capacity 

« 16GB of ECC memory (with the option to upgrade 
to 32GB) 

« 2x 1 Gigabit network controllers 

« Remote management port (IPMI) 

- Tool-less design; hot swappable drive trays 

« FreeNAS installed and configured 


Intel, the Intel logo, Intel Atom and Intel Atom Inside are trademarks of Intel Corporation in the U.S. and/or other countries. 


FREENAS 


CERTIFIED 
STORAGE 


With over six million downloads, 
FreeNAS is undisputedly the most 
popular storage operating system 
in the world. 


Sure, you could build your own FreeNAS system: 
research every hardware option, order all the 

parts, wait for everything to ship and arrive, vent at 
customer service because it hasn't, and finally build it 
yourself while hoping everything fits - only to install 
the software and discover that the system you spent 
days agonizing over isn’t even compatible. Or... 


MAKE IT EASY ON YOURSELF 


As the sponsors and lead developers of the FreeNAS 
project, ixsystems has combined over 20 years of 
hardware experience with our FreeNAS expertise to 
bring you FreeNAS Certified Storage. We make it 
easy to enjoy all the benefits of FreeNAS without 
the headache of building, setting up, configuring, 
and supporting it yourself. As one of the leaders in 
the storage industry, you know that you're getting the 
best combination of hardware designed for optimal 
performance with FreeNAS. 


Every FreeNAS server we ship is... 


» Custom built and optimized for your use case 

» Installed, configured, tested, and guaranteed to work out 
of the box 

» Supported by the Silicon Valley team that designed and 
built it 

» Backed by a 3 years parts and labor limited warranty 


As one of the leaders in the storage industry, you 
know that you're getting the best combination 

of hardware designed for optimal performance 

with FreeNAS. Contact us today for a FREE Risk 
Elimination Consultation with one of our FreeNAS 
experts. Remember, every purchase directly supports 
the FreeNAS project so we can continue adding 
features and improvements to the software for years 
to come. And really - why would you buy a FreeNAS 
server from anyone else? 


FreeNAS 1U 

- Intel® Xeon® Processor E3-1200v2 Family 

- Up to 16TB of storage capacity 

* 16GB ECC memory (upgradable to 32GB) 

« 2x 10/100/1000 Gigabit Ethernet controllers 
« Redundant power supply 


FreeNAS 2U 
- 2x Intel® Xeon® Processors E5-2600v2 Family 
- Up to 48TB of storage capacity 
- 32GB ECC memory (upgradable to 128GB) 
« 4x 1GbE Network interface (Onboard) - 
(Upgradable to 2 x 10 Gigabit Interface) 
« Redundant Power Supply 


a 


inside’ 
XEON® 


http://www.iXsystems.com/storage/freenas-certified-storage/ 


Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. 


EDITOR’S WORD 


Dear Readers, 
ust short summing up as it is summer time, and all of us need 
to have time for some fun. 


In the July issue of BSD Magazine, you have a chance to learn 
how to use Wireshark in a SAN environment. What is more, our 
expert will teach you how to create a Text Filled Vignette and 
how to manipulate images like a design professional. 


| recommend you read our Interview with Brett Davis. 
He will tell you more about iXsystems and TrueNAS, the enterprise 
appliance version of FreeNAS. 


Finally, you may find of interest the pkgng tool called “pkg” 
that is one of the modern and fast binary package managers. 
This tool was designed and developed for FreeBSD; however, 
PC-BSD used it in production first and was followed soon after 
by DragonFly. 


| would like to express my gratitude to our experts who 
contributed to this publication and invite others to cooperate 
with our magazine. 


The next issue of BSD Magazine will be published in 4 weeks. 
If you are interested in learning more about future content or, if 
you would like to get in touch with our team, please feel free to 
send your messages to ewa.d@bsdmag.org. | willbe more than 
happy to answer all your questions. 


Ewa Dudzic 
and BSD team 
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IN BUSINESS 


FreeNAS 


in an Enterprise Environment 


By the time you're reading this, FreeNAS has been downloaded 
more than 5.5 million times. For home users, it’s become an 
indispensable part of their daily lives, akin to the DVR. 
Meanwhile, all over the world, thousands of businesses 
universities, and government departments use FreeNAS to 
build effective storage solutions in myriad applications 


What you will learn... 


« How TrueNAS builds off the strong points of the FreeBSD and | 
FreeNAS operating systems 


* How TrueNAS meets modern storage challenges for entery 
he FreeNAS operating systems is fre 
the public and offers thorough doc 


| active community, and a feature-rig 
the storage environment. Based on Free 
can share over a host of protocols (SM§ 
FTP. iSCSI, etc) and features an intuiti 
the ZFS file system, a plug-in system 
much more. | 

Despite the massive popularity g 
aren't aware of its big brother dut 
data in some of the most demand 
environments: the proven, enterp 
professionally-supported line of 


But what makes TrueNAS diffa 
Well, I'm glad you asked... 


Commercial Grade Supp 
When a mission critical stor 
organization's whole operat 
halt. Whole community-bag 
free), it can't always get an 
and running in a timely 
responsiveness and expe 
dedicated support tea 
provide that safety. 

Created by the sa 
developed FreeNAS. 


Vj Simple Management 

(Vj Hybrid Flash Acceleration 
(VjfeIntelligent Compresssion 
(Vi All Features Provided 

Up Front (no hidden 
licensing fees) 


POWERED BY INTEL® XEON* PROCESSORS 
ntel, the lintel locpo, lintel Xeon and Intel Xeon Inside are trademarks of Irite 


Citrix makes and you recehve no representations of warranties 
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\ ( systems 


4, 


\ 
AN 


INTERRUPT THIS MAGAZINE TO BRING 
YOU THIS IMPORTANT ANNOUNCEMENT: 


THE PEOPLE WHO DEVELOP FREENAS, THE WORLD'S MOST 
POPULAR STORAGE OS, HAVE JUST REVAMPED TRUENAS. 


POWER WITHOUT CONTROL MEANS NOTHING. 
TRUENAS STORAGE GIVES YOU BOTH. 


Mi Self-Healing Filesystem 

Mi High Availability 

(Vie Qualified for VMware and 
HyperV 

Vi Works Great With Citrix 
XenServer® 


To learn more, visit: www.iXsystems.com/truenas 


nother countries. 
ware, Inc. In he United States and other jurisaictiors. 


5, its functianality, the test(s) of the results 


event shall Citrix be liable for an i dam ages ot any kind whatsoever arising Out 


le. punitive or other damages. 
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Tools 


OS DragonFlyBSD Package Management 
System — dports 
Siju George 
The pkgng tool called “pkg” is one of the modern and fast 
binary package managers. This tool was designed and 
developed for FreeBSD; however, PC-BSD used it in 
production first and was followed soon after by DragonFly. 
In the future, it will be the only binary package manager on 
FreeBSD, just like it is the only port manager in DPorts. 
Siju will present you with the most useful options and 
direct you if you need more. 


Graphic Designing 
12 Getting to Grips With the Gimp 


Rob Somerville 
For those of you who want to learn more about GIMP and 
it is features and make some useful improvements to your 
Family Album, we present the sixth part of Rob’s series. 
In this part in our series on the Gimp, Rob will teach you 
how to create a Text Filled Vignette and how to manipulate 
images like a design professional. 


security 


=O Network Analysis On a Storage Area 
Network Using Wireshark 
Sembiante Massimiliano 
Wireshark, originally known as Ethereal, is probably the 
most famous open source packet sniffer and network 
analysis tool available. This application supports about 1300 
protocols through a vast number of filters. Functionalities 
such as traffic, protocol analysis, and packet dissector make 
it an extremely versatile tool for security experts, network 
engineers, and system administrators. Sembiante will 
tell you how to use Wireshark in a SAN environment. 


=< Penetration Testing Projects: Memory 
Traces of the Last 9 Years 
Mrityunjay Gautam 
In the last five years, Mrityunjay has worked on a series 
of application penetration testing projects on a very wide 
variety of products. Mrityunjay had a chance to look 
at kernel components in Unix environments, antivirus 
products, storage products, compliance products, mobile 
products, cloud products, web based products and 
virtualization products. 
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Tips&Tricks 


=S Auto-install Feature Using a Support CD 


Wesley Mouedine Assaby 
Wesley, in his article, will teach you how to provide an 
answer file for the auto install process. 


DataBases 


30 Database Query Optimization for Huge 
Databases 
Mark Sitkowski 
We have the opportunity to give the database engine a 
helping hand, and improve the performance of a long- 
running SQL query. We do this, by not performing the 
whole query in SQL. We already know better than to 
perform ORDER BY in SQL (see ‘Embedded SQL.doc’), 
now we shall see how to speed up certain types of query... 


36 Inserting and Retrieving Data From 
Huge Databases 
Mark Sitkowski 
The focal point of our attention, is the database, from 
which and into which we wish to transfer extremely large 
quantities of data. When we invite our database engine 
to execute a SELECT statement, two things happen. 
Find out more! 


Spotlight 


4.2 Interview of the Month 
Brett Davis Tells You All About 
TrueNAS 


Luca Ferrari 
Column 


<1G Is There a Down Side to Mass 
Connectivity? 
Rob Somerville 


Meet Our Expert 


48 Interview with . 
Antonio Francesco Gentile 
BSD Team 
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CYBER 
SECURITY 


A N EW event, 


ExCeL. London for a new era of cyber threats 


www.cybersec-expo.com 


» The most comprehensive analysis anywhere of how to protect 
the modern organisation from cyber threats LE = 


» Free to attend seminars delivered by Mikko Hyfponen, 
Eugene Kaspersky and many more 


» Attend the “Hack Den” a live open source security lab to 
share ideas with White Hat hackers, security gurus, 
Cyber Security EXPO speakers and fellow. professionals 


» Network with industry experts.and meet with Cyber 
Security exhibitors 


» Discover what the IT Security team of the future 
will look like | 


Cyber Security EXPO is the new place for everybody wanting to protect 


their organisation from the increasing commercial threat of cyber Co-located at 

attacks. Cyber Security EXPO has been designed to provide CISOs and IP O EUROPE 
IT security staff the tools, new thinking and policies to meet the 21st -9 October 2014 ExCeL. London 
century business cyber security challenge. 


www.ipexpo.co.uk 
Cyber Security EXPO delves into business issues beyond traditional 

enterprise security products, providing exclusive content on behaviour 

trends and business continuity. At Cyber Security EXPO, discover how 

to build trust across the enterprise to securely manage disruptive 

technologies such as: Cloud, Mobile, Social, Networks, GRC, Analytics, 

Identity & Access, Data, Encryption and more. 


networks 


hint Ap i ee es ye 
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ME paloalto 5 | Schneider \vVeecam vymware 
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DragonFlyBSD Package 


Management System 


— dports 


The recent releases use dports based on the FreeBSD ports 
collection unlike the older releases which used pkgsrc from 
the NetBSD project. The pkgng tool called “pkg” is one of the 
modern and fast binary package managers. This tool was 
designed and developed for FreeBSD; however, PC-BSD used it 
in production first and was followed soon after by DragonFly. 
In the future, it will be the only binary package manager on 
FreeBSD, just like it is the only port manager in DPorts. 


What you will learn... 
« What dports is 
« How to deal with dports 


n order to install dports on your system, use the follow- 
ing commands: 


' Od 7 uee 
# make dports-update 


If /usr/local/etc/pkg.conf exists remove it 
# rm /usr/local/etc/pkg.conf 


if /usr/local/etc/pkg/repos/df-latest.conf does not ex- 
ist then create it 


# cd /usr/local/etc/pkg/repos && mv df-latest.conf.sample 
dit-lLacest. cont 


In order to install third-party software, you need to use 
the following commands: 


BSD 


What you should know... 
« FreeBSD Basics 


# pkg update 
# pkg install “software” 


Working with dports in detail 

First if Update pkg repository, search and _ install 
a package. 

# pkg update 


Updating repository catalogue 


00:06 
00:43 


100% 989KB 164.8KB/s 278.8KB/s 
100% 4372KB 101.7KB/s 261.9KB/s 


digests.txz 


packagesite.txz 


Incremental update completed, and 21191 packages 
processed: 


20894 packages updated, 139 removed and 230 added. 
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DragonFlyBSD Package Management System - dports 


# pkg search nginx 


noinx=1.. 6.072 
nginx-devel-1.7.0 


# pkg install nginx-1.6.0,2 
Updating repository catalogue 
The following two packages will be installed: 


Upgrading pcre: 8.34 -> 8.34 1 [Avalon] 
Instelling nginx: 1.6.0,2 [Avalon] 

The installation will require 715 KB more space 
1 MB to be downloaded 

Proceed with installing packages [y/N]: y 


pere-3.34 1.,txz 


100% 1039KB 94.4KB/s 58.8KB/s 00:11 
none l.6,0;,2.0%2 
100% 258KB 128.9KB/s 197.8KB/s 00:02 


Checking integrity... done 


[1/2] Upgrading pcre from 8.34 to 8.34 1... done 

[2/2] Installing nginx-1.6.0,2...===> Creating users and/ 
or groups. 

Using existing group ,www’. 

Using existing user ,www’. 


Now it is done. 


Removing a package from the System 


# pkg delete nginx 
Uninstallation has been requested for the following 
packages: 


noinst=1 96.052 
The uninstallation will free 683 KB 
Proceed with uninstalling packages [y/N]: y 
[1/1]. Deleting noinx=1.6.0,;2.s6 done 
You can always delete packages which were automati- 
cally installed as dependencies especially if you do not 
require them anymore. 


# pkg auto remove 
Uninstallation has been requested for the following two 
packages: 


argp-standalone=1.3.2 
mysqlSo-Clrent=5.5.07 
The uninstallation will free 36 MB 
Proceed with uninstalling packages [y/N]: y 
[1/2] Deleting argp-standalone-1.3 2... done 


[2/2] Deleting mysql55=client=5.5237 2:4 done 


www.bsdmag.org 


Now, you can list all installed packages on the system 


and get more details about them. 

#pkg info 

ORBIt2=2.14,19 High-performance CORBA ORB 
with support for the C language 

Phune r= 1.26.51 Xfce file manager 

appres-1.0.4 Program to list 

application’s resources 


atk 22 840 
(ATK) 


GNOME accessibility toolkit 


avahi-app-0.6.31.1 Service discovery on a 


local network 


Dasha 3.08 The GNU Project’s Bourne 
Again Shell 
bind9s=9..8../ BIND DNS suite with updated 


DNSSEC and DNS64 
Dbitmap= Ls) Bitmap editor and converter 
utilities for X 
Ga. Toct Niss=7.15.5 The root certificate bundle 
from the Mozilla Project 
CalroalslU.2 872 Vector graphics library 
with cross-device output support 
Catitarell—fonte=0.0.15 Cantare 


~------------- Output Curtailed ------------------ 

2.5: Display current version of ‘pkg’. 

# pkg -v 

Lo 2 

2.6: Audits installed packages against known vulnerabilities. 
# pkg audit -F 


VuUliesmb. bz 
100% 441KB 220.8KB/s 
dbus-1.6.18 is vulnerable: 
dbus = local Dos 
CVE: CVE-2014-3477 
WWW: http://portaudit.FreeBSD.org/52bbc7e8-f13c-1le3-bc09- 
beaec565249c. html 
See OULOU CUrreaLled. == =—S.=5S5=5>5S5Ss=se5 


51.5KB/s O07 0z 


Upgrade all packages on the system. 
# pkg upgrade 


Updating repository catalogue 
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digests. txzZ 


100% 989KB 27.5KB/s 48.8KB/s 00:36 
packagesite.txz 
100% 4372KB 37.7KB/s 11.9KB/s Q1256 


Incremental update completed, and 21191 packages processed: 
20894 packages updated, 139 removed and 230 added. 
Upgrades have been requested for the following 25 packages: 


Installing openssl: 1.0.1.13 [Avalon] 

Upgrading ca root nsé: 3.15.4 => 3.16 [Avalon] 

Upgrading. curl: 7239.0 => 7.236201. [Avalon] 

Upgrading gettext: Usles3.1 => U.18.3:1 1 [Avalon] 
etait OUTDO. "(CUtad Wed, ===a=== 45S ees aS ae 


Installing perl5s 5.16.3. 10 - [Avalon] 
Installing p5-Digest-HMAC: 1.03 [Avalon] 
Upgrading git? 1.8.5.4 —> 1.9.3 [Avalon] 


The upgrade will require 131 MB more space 
42 MB to be downloaded 

Proceed with upgrading packages [y/N]: y 
openssl-1.0. 1 13 .txz 


100% -2542KB. 32.2KB/s 2l28KB/s 01219 
Ca TOOt 1Ses=3. 16 ste 
100%. 301KB O60.IKB/s S0./7KB/Ss 00205 
curl=7.956.0 L.itez 
100; T248KR -33:7KB78 ITI. 7KB7s. 00237 
eats aia OUR OUiC Cur ted ed \=S==—=3=——=s=s5—45-=555> 
[25/25] Upgrading git from 1.8.5.4 to 1.9.3...===> 


Creating users and/or groups. 

Using existing group ‘git daemon’. 

Creating user ‘git daemon’ with uid ‘964’. 

Updating /etc/shells 

done 

Copy /usr/local/openssl/openssl.cnf.sample to /usr/local/ 
openssl/openssl.cnf 

and edit it to fit your needs. 


Bene a Se OUL DUE. CUttetleg Seas SS 2S Ses SSeS 
Clean the local cache of fetched remote packages. 
# pkg clean 


The following package files will be deleted from the cache 
directory 


/var/cache/pkg: 

Package: Origin: Reason: 
All/tmux-1.9.a 1.txz sysutils/tmux Size mismatch 
All/bash-4.3.8.txz shells/bash 


superseded by bash-4.3.18 2 
Sa a a ce Out pi Cube led sea =a=SSS5S=S==S 
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All/mime-support-3.54.txz misc/mime-support 
Checksum mismatch 
AlLl/cmdwatch=-0.2.0 2.txz sysutils/cmdwatch 


Checksum mismatch 


Proceed with cleaning the cache [y/N]: y 

Deleting: 
/var/cache/pkg/All/tmux-1.9.a_1.txz 
/var/cache/pkg/All/bash-4.3.8.txz 
/var/cache/pkg/All/libevent-1.4.14b 3.txz 

a ea OULU CUPGeLed’ SSeSSeeSsS= S54 === 
/var/cache/pkg/All/mime-support-3.54.txz 
/var/cache/pkg/All/cmdwatch-0.2.0 2.txz 

All done 


Conduct sanity checks for installed packages 
# pkg check -s -a 

pkg: fopen(/usr/local/etc/smartd.conf.sample): No such file 
or directory 


smartmontools-6.2 2: checksum mismatch for /usr/local/etc/ 


smartd.conf.sample 


Query the database for package(s) that installed a spe- 
cific file 


# pkg which /usr/local/bin/rsync 
/usr/local/bin/rsync was installed by package rsync-3.1.0 3 


Conclusion 
Dports is a versatile package management system. If you 
want to learn about the options, you need to refer to 


# man 8 pkg 

on your own dragonfly system. Linux users using popu- 
lar distros will find many similarities in dports with their 
package management systems such as apt, yum, pac- 
man, emerge, etc. 


SIJU GEORGE 


Siju George is Senior Systems Administrator at HIFX IT & Media Servic- 
es Private Limited. Experienced in Systems Administration on Open- 
BSD, FreeBSD, DragonFlyBSD, Debian, Redhat and other flavors of 
Linux, Mac OS X and Microsoft Technologies since 2002. 
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On FreeBSD® & PC-BSD® Merchandise 


229.95 


PLB 9.1 Ot 


239.95 


Pree) 9.1] evaded Carke CD Set 
mr Freenob 9.1 Cl 


/ Shyla Dre Atture z 


Ys 
nis 


F. 
et 


FreeBSD 9.1 Jewel Case CD/DVD... $39.95 


CD Set Contains 
Diet) installation Boot Lined 1356) 
Gisc? Exeential Packaged Korg (38d) 
Die 3 Esnental Packages, GMOME? (idi6! 
ier Extential Packages (| 284) 


Peet S60 Co ee ee 934995 
Fe is 995 
FreeBSD Subscriptions 

Save tine and $55 by subscribing to regular updates of FreeBSD 
Frees Subscription, start wits C10 9,1 ,.ssssssssssessssses ines il Ds oP 
FreeiSD Subseription, start with DVD G1... csesceseseeenees $29.95 
FreeiSD Subscription, start Wit C0 G0 sccsssssssssssssssssssssssssoas Sal 
FreeiS0 Subscription, start with DVD 9,10........ssceccsssserererere §279.95 
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The FreeBSD Handbook 
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The FreeBSD Handbook, Volume 2 (Admin Gulede).............. $39.95 


The FreeBSD Handbook Specials 
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The FreeBS0 Handbook, Both Volumes & FreefSD 9-1 ........ 579.95 
PC-BSD 9.0 Users Handbook _...00 $24.95 
BSD Magazine... See eeryeyerperrereerenrerrrererteyer, $11.99 
The FreeBSD Toolkit DVD $39.95 
FreeBSD Mousepad S10.00 
FreeBSD Ge PCBS Cap nnn sccasccnscnecneenees 420.00 
BSD Dimmers Pr ccc ctcccssenee $2.00 


For even MORE items 
visit our website today! 
www. FreeBSDiViall.com 
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Getting to Grips With 
the Gimp - Part 6 


A Text Filled Vignette 


In the sixth part in our series on the Gimp, we will learn 
about creating a Text Filled Vignette. 


What you will learn... What you should know... 
« How to manipulate images like a design pro « General PC administration skills 
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3 he book, “Tinker Tailor Soldier Spy” by Jc h 


Le Carre has a vignette of George Smiley on— 
the front cover. Hence, we will create a simil ir 
edgy picture with text rather than numbers. | 
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Getting to Grips With the Gimp - Part 6 


The recipe 
We will follow the following steps as indicated: 


Step 1 
Download the image of the fe- 
male model from Table 1. 


Details and Credits Image 


Joann p 02 portrait Female model http://www.freeimages.com/photo/668970 
Uploaded by obyvatel 


Step 2 
Pull the guide down from the top of the 
measuring bar, so that it rests just be- 
neath the model's right eye [Screen- 
shot 1]. 
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GRAPHIC DESIGN 


Step 3 

Using the rotate tool, rotate the image 
until the eyes become parallel to the 
guide [Screenshot 2]. 


Step 4 ° oeantpry 
Using the move tool, bring the layer PR, weaken. | 
down until the chin is at the bottom of g= =" "ea 4 
mie Sole 
the frame. Use the crop tool to cen- eT. 
tre the face by removing excess areas | 
[Screenshot 3]. - eto 
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Getting to Grips With the Gimp - Part 6 


Step 5 

Use the clone too/ to the area of the 
image left transparent due to the ro- 
tation. Use a large size brush to re- 
create the hair on the left hand side 
[Screenshot 4]. 


Step 6 
Zoom in each eye in turn and carefully 
select around the iris. Use the shift key 
to select the area around the second 
eye. Click on the paths tab and choose 
selection to path [Screenshot 5-6]. 
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Step 7 

Add a new layer, pick a color for 
the eyes and fill the selected areas. 
Change the layer mode to dodge 
[Screenshot 7]. 


Step 8 

Using the smudge tool, touch up 
around the eyes dragging the col- 
or away from the edge of the iris, so 
that no sharp edges still remain. Add 
a new layer and fill with a 45 degree 
gradient from top left to bottom right. 
[Screenshot 8]. 
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Getting to Grips With the Gimp - Part 6 


Step 9 

Add a new layer as in Step 8, and 
change the mode to dissolve. Adjust 
the opacity until you get the level of in- 
terference you like. Similarly, change 
the layer in Step 8 until you get an ef- 
fect you prefer. In the final version, | 
chose burn [Screenshot 9]. 


Step 10 

Using the text tool, select the whole 
image and then enter the text you 
want as the message. This is very 
processor and graphics intensive, but 
Sans 5Opt seemed to work OK on my 
elderly PC. The smaller the font, the 
more text and the slower this opera- 
tion will be. Change the layer mode to 
overlay mode [Screenshot 10]. 
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Step 11 

Using the erase tool, remove all the 
text detail and gradients from the in- 
side of the eyes. Experimenting with 
the text layer, duplicating it and chang- 
ing the mode then trying to delete text 
from the face is a good technique. 


Step 12 
The final result that | saved will be 
similar as in [Screenshot 11]. 


ROB SOMERVILLE 


Rob Somerville has been passionate about technology since his early teens. A keen advocate of open systems 
since the mid-eighties, he has worked in many corporate sectors including finance, automotive, airlines, govern- 
ment and media in a variety of roles from technical support, system administrator, developer, systems integra- 
tor and IT manager. He has moved on from CP/M and nixie tubes but keeps a soldering iron handy just in case. 
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Performance and 
Reliability is critical 
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Network Analysis 


On a Storage Area Network Using Wireshark 


Wireshark, originally known as Ethereal, is probably the 
most famous open source packet sniffer and network 


analysis tool available. 


What you will learn... 
¢ What Wireshark is 
¢ How to use Wireshark in a SAN environment 


through a vast number of filters. Functionalities 

such as traffic, protocol analysis, and packet dis- 
sector make it an extremely versatile tool for security ex- 
perts, network engineers, and system administrators. 


Te application supports about 1300 protocols 


Wireshark can be used during a proactive analysis to 
identify potential network bottlenecks, to monitor “live” 
what is happening to data flow, and to decode packets in 
transit, displaying information in readable format. The tool 
can be installed on any computer connected to the net- 
work and equipped with a NIC card. Using specific API or 
libraries, such as WinPcap under Windows or libpcap for 
Unix, it enables data capture and allows analysis of pack- 
ets travelling over the carrier. 


Commonly, Wireshark is used on Ethernet technolo- 
gy or Wireless networks, but its also possible to use it 
for SAN (Storage Area Network) to analyze FCP (Fiber 
Channel Protocol) over Optical Fiber Cables. 


The Storage Area Network Architecture 
SAN (Storage Area Network) is generally defined as a 
dedicated storage network using Fibre Channel technol- 
ogy to provide disk volumes on the target host. 


The SAN environment can be designed to have a disk 
array directly attached to a host or through a SAN Switch 
(a SAN Network Director similar to an Ethernet Switch) in 
order to connect multiple hosts to a single array and enable 
Business Continuity and Disaster Recovery capabilities. 
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What you should know... 


¢ Security Basics 


Disks’ capacities are presented as logical volumes 
called LUNs (Logic Unit Number). The provisioning is per- 
formed by connecting the Array, Switch and HBA (Host 
Bus Adapter, a fiber card adapter installed on the Host 
system) using two different operations called LUN Mask- 
ing and Zoning (Figure 1). 


With Zoning, we connect the ports of the devices, also 
called initiators, to be logically linked. While performing 
the LUN Masking, we present the LUN (disk capacity) to 
the target host. 


The SAN directors are accessible by Storage and Net- 
work Administrators via the Terminal Access Controller 
Access-Control System (TACACS) or Remote Authenti- 
cation Dial In User Service (RADIUS). 


Server Syeherris 


i 
Fc Cables 
i 


Storage Systems 


Figure 1. Fiber Channel Zoning 
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Network Analysis On a Storage Area Network Using Wireshark 


The main difference between NAS and SAN volume 
provisioning systems is the protocol used to provide stor- 
age capacity. NAS uses NFS or CIFS protocols while 
SAN uses the FCP (Fiber Channel Protocol). 


Fiber Channel Protocol 

The FCP (Fibre Channel Protocol) is a transport pro- 
tocol similar to TCP/IP, approved as ANSI stan- 
dard around 1994. FCP mainly transports SCSI com- 
mands using the Optical Cable as a carrier (Figure 2). 
This protocol was invented to enable higher perfor- 
mances and distance insensitivity, to facilitate the sys- 
tem boot from external devices, and to support enter- 
prise storage flexibility and scalability. 


Fiber Channel Traffic Analysis 

Network analysis on a fiber channel is not the same 
as over the Ethernet. There's no equivalent promiscu- 
ous mode for nodes, so you can’t listen to traffic moving 
through the network. To achieve traffic analysis, first of all, 
you need to tap into the network between the source and 
destination ports you wish to analyze. Dedicated hard- 
ware is necessary to “read” the packets and specific soft- 
ware to analyze the frames. 


some examples of external frame analyzers are: Xgig 
Protocol Analyzer Family from JDSU or LeCroy FC Pro- 
tocol Analyzers. 


FC frame analyzers are often accompanied by dedicated 
TAP (Traffic Access Point) network hardware. This device 
is physically inserted into the network and when turned 
on, it copies all frames headed for a specific port to a spe- 
cific TAP port. Using TAP hardware means that the frame 
analyzer can be plugged into the TAPped port and then 
removed without causing an interruption in the FC net- 
work flow. Of course, in order to initially install the TAP 
hardware, you have to interrupt the network flow. 


Preferably, these devices should be permanently con- 
nected because each time you insert and remove the ana- 
lyzer, you interrupt the FC network flow. This may result in 
serious repercussions for the system, such as Data Loss 
and Kernel Panic. 


In some cases, this has been made easier by vendors 
such as Cisco and Brocade, providing a Switched Port 
Analyzer (SPAN) feature, which copies most traffic going 
to a specific port to another switch port called “mirror port”. 
In that case, the frame analyzer or PAA (Protocol Analyzer 


Figure 2. Fiber Cable 


Figure 3. Typical SPAN to PAA Configuration 
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Adapter) can be plugged into the SPAN switch port and 
analyzes the traffic flow (Figure 3). 


Cisco and Brocade provide native command line tools 
to allow local fiber channel control traffic passing through 
the local supervisors to be copied into a text file that is 
stored in a chosen location on the switch or redirected to 
the IP Address. 


The default behavior is to store the output in a volatile 
storage area. This can later be copied to a remote server 
for analysis with Wireshark. 


It is also possible to specify a remote IP address to send 
the data to, and Wireshark can be used to analyze the 
data in real time, as It’s collected. 


Cisco MDS Switches with the SanOS operating system 
provide an FC Analyzer command line called: fcanalyzer 
(portlogshow is the command line on brocade). 


In order to configure the system to perform traffic analy- 
sis, we must configure the Switch in passive remote mode 
using the command line as follows: 


Figure 5. Remote Connection via Command Line Interface 
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Figure 6. Host Login Trace 
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MDS3 (config) # fcanalyzer remote 172.xxx.xxx.Xxx 
MDS3 (config) # exit 

MDS3# show fcanalyzer 

PassiveClient = 172.xxx.xxx.xXxx 


MDS2# 


Next, we instruct Wireshark to connect to it remotely us- 
ing the graphical interface (Figure 4). Or, we may try to 
connect to it using the Wireshark CLI (Figure 5). 


Now, we are ready to start a new capture session 
and verify which type of raw data we can get out of the 
FC analyzer. 


Wireshark can capture a huge amount of information, 
when installed between the disk array and the host ma- 
chine. It could potentially intercept all the SCSI com- 
mands passing through these two devices. At the same 
time, it is possible to inspect what is happening at the 
switch level and use the data for troubleshooting and de- 
bugging purposes. 


During a live-capture session, we can monitor the 
Fabric behavior and the Zone-sets operations; or, we 
can display which initiators and nodes were currently 
active and enabled. It is possible to verify volumes pre- 
sented to the hosts and potentially reverse engineer the 
entire SAN configuration. 


We can manage to identify all the Zoning and Masking 
setup, and if the Switch is using features such as VSAN 
(Virtual SAN similar to VLAN in Ethernet Networks) or IVR 
(Inter-VSAN Routing), we can trace all the members’ de- 
vices existing in all of the SAN areas including all the SC- 
S| command dialogs. 


With the help of customized filters, it is possible to use 
Wireshark for troubleshooting purposes and display (for 
example, merge conflicts, Fabric Login status, Zoning 
failure, and so on). A good example is visible in Figure 
6. We can see a live capture session with Wireshark 
tracing a Host Login event. It is possible to trace the 
entire “dialog” between the Host and the Remote Ar- 
ray through the Switches. There are two active windows 
in Wireshark: 


¢ Transmit Trace 
e Response Trace. 


The first one is tracing FCP/SCSI transmission dialog 
and the second traces the responses. 
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Appendix 1 
http://en.wikipedia.org/wiki/Fibre_Channel 


http://en.wikipedia.org/wiki/Fibre_Channel_Logins 
http://en.wikipedia.org/wiki/Fibre_Channel_zoning 


http://www.brocade.com/products/all/switches/index.page 


http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/4_1/configuration/guides/cli_4_1/tsf.html 


http://www.jdsu.com/en-us/Test-and-Measurement/Products/a-z-product-list/Pages/xgig-protocol-analyzer-family-overview.aspx 
http://teledynelecroy.com/protocolanalyzer/protocolstandard.aspx?standardid=5 


http://www.cisco.com/en/US/products/hw/ps4159/ps4358/products_configuration_example09186a008026eb55.shtml 


In the first window, we can see LUNs (remote disks) are 
in “inquiry status” (seeking to log on to target host) and 
the FC initiator is attempting to initiate the FLOGI (a link 
service command that sets up a session between two par- 
ticipants’ devices). 


We can verify the positive response in the second win- 
dow. The Login request is accepted, and we can see the 
positive response. The trace window is now displaying 
that LUNs are reported in good status, hence available to 
be mounted on the target Host. 


Conclusions 

This article provides a quick overview of using Wireshark 
In a SAN environment. Although network analyzers are 
powerful software and can be used to troubleshoot com- 
plicated issues, at the same time they can be extremely 
dangerous when misused or activated through unauthor- 
ized access. 


Sniffers are difficult to detect and can be applied almost 
anywhere within the network under analysis, which makes 
them one of the hackers’ favorite tools. 


We need to bear in mind that NO Firewalls or IDS are 
present in a SAN environment; thus, it is not possible to 
filter traffic or identify intruders easily. 


The Login of a “new” device in the fabric is never reported 
as malicious activity and poorly monitored. Moreover, a vol- 
ume can be mounted and shared over multiple hosts and, 
in most cases, there is no event alert that traces the activity. 


It’s true that the SAN protocol presents all data at the 
block level, but it is still possible to capture and dump, ina 
separate storage area, a large quantity of traffic to attempt 
file reconstructions later. 


Using Wireshark to perform SAN network cartography 
may be a good starting point to perform further attacks. 
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One may be able to use the information gathered to re- 
configure Zoning and Masking, mount the target volume 
on a different Host, and access the stored data. 


FCP is a protocol that does not provide encryption; thus, 
all the data travelling is potentially exposed. 


Remember to handle all the information gathered with 
Wireshark carefully in order to avoid data leakage. We 
should store all the captured files securely, possibly in en- 
crypted volumes and never forget that sniffing is an illegal 
activity when performed without authorization. 
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Projects 


Penetration Testing 


Memory Traces of the Last 5 Years 


In the last five years, | have worked on a series of application 
penetration testing projects on a very wide variety of 
products. | have a chance to look at kernel components in 
Unix environments, antivirus products, storage products, 
compliance products, mobile products, cloud products, web 
based products and virtualization products. 


What you will learn... 
« How to deal with the Unix environment 
¢ How to select the best Pentester’s applications 


ing, | have seen a large number of minor as well as 

major security issues in these products. In this article, 
| would like to talk about some of the very interesting se- 
curity issues that | found doing these pen tests, along with 
the correct way to fix them. 


n the course of my time spent doing penetration test- 


Please note that all the vulnerabilities being discussed 
here have already been fixed after the pen test report was 
submitted (hopefully, there are no regressions in the later 
versions) and hence, do not expect to find a zero-day in 
this article. The article is intended to give an idea to the 
readers about what all can go wrong besides the things 
shown by the standard pen test tools or fuzzers, and what 
to look for when you encounter a similar application for 
the pen test. 


“- Bug 1 


‘eset? 

Stored XSS from Intranet (CVSS = 7.3) 
(AV:A/AC:L/Au:N/C:C/I:P/A:P) 
lt was a bug | found during a penetration test of a scan- 
ning server. The architecture dictated that the client 
needed to send a file name over the LAN, and this scan- 
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What you should know... 


¢ Security Basics 


ning server would then let you know the scan status for 
this file. The activity was logged in the log files of the 
scanning server. This scanning server had a web man- 
agement interface, which was accessible by the admin 
of the scanning server. From the web interface, he/she 
would be able to see the log files which contained the 
following information; time-stamp; name of the file; ac- 
tivity log. 


Here, the name of the file was something that was ex- 
ternally controlled. Hence, the attack, which would work 
here, was to send a request for scanning to the scanning 
server with the file name as a JavaScript code. Now, due 
to the logging activity, the script would be logged as is on 
the log file since it is treated as a filename. Now, when- 
ever the administrator would log on to the web console, 
this script would get activated and steal the cookies for 
the attacker. 


This was a persistent XSS attack, but since this required 
us to send a TCP packet directly to the server, which is 
positioned inside the network, the range of this attack was 
Intranet and not Internet, as is the case for most of the 
other XSS attacks. 
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| found this attack particularly interesting because, un- 
like most of the XSS attacks, this was something which 
didn’t need any action on the part of the victim, and it was 
not entirely a web-based attack. This was a network at- 
tack, which targeted the web interface for session hijack. 


Bug 2 


ActiveX Control Design Flaw 

(CVSS = 7.8) (AV:N/AC:L/Au:N/C:C/I:N/ 
A:N) 

| found this bug while looking at a lab management prod- 
uct. This product was supposed to manage a large num- 
ber of computers by putting an agent on these machines. 
Along with the agents, it would also install a bunch of Ac- 
tiveX controls that was not set with Kill Bits. For those 
of you who do not know what is a kill bit or ActiveX con- 
trol, here is a brief description. An ActiveX control can be 
roughly seen as a COM object, which exposes a bunch of 
APIs, which are accessible over the web. Any web page 
can call on these APIs and run the COM object on the 
local machine. This can act as an entry point from the 
browser to the actual machine. 


f 
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Kill bit is a flag, which can be set in the Windows reg- 
istry. Once set, it disallows the ActiveX control from run- 
ning on the Internet Explorer or equivalent interfaces. 
Hence, this is the easiest way to disable the ActiveX con- 
trol from being called from the web pages. 


Coming back to the vulnerability, this ActiveX control 
had an API called RunNotepad. This function would take 
one parameter, which was intended to be a file name. 
The effect was that it would open that file in the Notepad 
application on the local machine. Looking for the possible 
avenues of attack, | tried fuzzing the string input to this 
API and calling this API a large number of times. | even 
tried command injections to start another binary, but this 
led to nothing. If it had, | would not be writing about this 
bug here, would |? 


Now, looking beyond the obvious, | explored the vari- 
ous command line switches that Notepad could take. It is 
interesting to see what /p could do for notepad. If we run 
notepad /P password.conf, It would redirect the password. 
conf file to the default printer without any requirement for a 
user interaction and then it would close the notepad appli- 
cation as well. Hence, this string worked like magic when 
used as the attack parameter. In our regular day-to-day 
usage of notepad, we never use switches on this applica- 
tion. It is very interesting to see what all the switches can 
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do even when the product has been developed correctly 
with all the checks for buffer overflows, shell metacharac- 
ters, format strings, etc. 


Bug 3 


Setuid Root Binary on Unix 

(CVSS = 7.1) (AV:A/AC:L/Au:S/C:N/I:C/A:C) 
When you are dealing with an appliance or any product 
where you are pen testing a kernel component, you would 
typically find a lot of user land binaries which are intended 
for calling in the kernel functionalities. One of the most 
commonly found binary types is a setuid binary. 


s 
es 
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A setuid binary is an executable on Unix which can 
be called by any user on the machine, but it would al- 
ways run with the privileges of the owner of the binary. 
This gets particularly interesting when the owner of the 
binary file is root itself. Now, when the binary is called, it 
starts running with root privileges. Since there is a clear 
privilege escalation, the setuid root binaries are typically 
self-contained, and they take nothing but the command 
line parameters as input. This input is very closely sani- 
tized and then consumed in the binary. A very commonly 
used example of such a binary is the passwd command. 
This binary has to run as root because it has to write to the 
/etc/shadow file in the Unix file system which is root read- 
able only. | was looking at one such setuid binary for a 
Unix product, and | tried all kinds of command line fuzz- 
ing on it. The binary never crashed, and if there were 
an unexpected parameter, it would safely throw an error 
message and exit. | must mention here that if you ever 
see any setuid root binary throwing an error message, 
this in itself should ring a bell in your head that this may 
be a vulnerability. 


At this point, | would like you to recall the concept of I/O 
Redirection, which is present in all operating systems, in- 
cluding Unix. Hence, if we redirect the output of this setuid 
root binary to /etc/passwa file, and redirect the stderr to 
stdout, this would ensure that the error message would 
now get written in the /etc/passwd file. Please remember 
that this action would overwrite the passwd file and hence, 
no user would be able to login into this machine anymore. 
The exact command to do this would be: 


$ ./setuid binary (incorrect param _to get error message) > 


/etc/passwd 2>&1 


This issue is a bit tricky to fix because there are bound 
to be setuid root binaries in the Unix systems, and if 
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there are incorrect inputs, they are bound to return some 
error message. The correct way to go about fixing this is 
to drop privileges to the userid of the caller before writ- 
ing anything to any open file descriptor, including stdout 
and stderr. 


“- Bug 4 


‘eae? 
( § Authentication Bypass (CVSS = 8.3) 
(AV:A/AC:L/Au:N/C:C/I:C/A:C) 
One of the biggest mistakes committed by software engi- 
neers is to introduce proprietary authentication modules 
or add proprietary encryption functions with the hope 
that since the protocol or the algorithm is not known to 
the general public, they can never break into this. | must 
emphasize that this is strictly an incorrect assumption. 
| would quote one such example here where | was sup- 
posed to pen test a web application. This web application 
had a proprietary authentication module over the network 
that functioned as follows: 


¢ Client would send a TCP Connect Request to the 
server 

¢ Server would respond with a challenge message af- 
ter creating a randomly generated file name. For ex- 
ample, it would create a file called baivinrg9j34fnger 
n9ng43invsrb93q4vib9r In the webroot folder. 

¢ The client is expected to give the name of this file as 
a response to the server's challenge message. 

¢ When the server receives a response, it will check if 
the file exists. If yes, it would delete this file and al- 
low access to the client. For the next authentication, it 
would create another file name; hence, the response 
would be new. 


As you all are correctly thinking, the client can never 
send the correct response unless and until, the client is 
on the same machine as the server, and it has access to 
the webroot folder to figure out the name of this random- 
ly generated file. Well, in the case of this product, this 
was the case. Hence, the first question to be asked here 
was — why do we need to open this socket on all inter- 
faces? We could have just opened it on 127.0.0.1 (local- 
host), if all that was required were an inter-process com- 
munication channel over the socket. 


This was a trivial flaw compared to what is a significant 
design flaw in this case. All that the server was doing was 
to verify the existence of the file whose name was sent by 
the client as the response message. Hence, if the client 
wanted, it would have sent the name of any file from the 
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webroot, and it would have worked. So how about /ogin. 
php?? And yes, it actually worked !! 


So, combined with the flow above of opening a sock- 
et on all interfaces, this attack became an intranet based 
attack where anyone could authenticate to the server by 
connecting to the server and sending the name of any val- 
id file from the webroot. And yes, the cherry on the cake 
was that the file would get deleted from the webroot, caus- 
ing a denial of service to the web users. 


As an add-on, to make matters worse, the file name 
here was not restricted to the webroot but it could have 
been any arbitrary file name in the file system. This serv- 
er was vulnerable to directory traversal attack as well. 
Hence the response from the client side could have been 
../../../../../../../etc/passwa and the server would 
happily delete this file and make the machine inaccessi- 
ble henceforth. 
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TIPS&TRICKS 


uto-install Feature 
sing a Support CD 


What you will learn... 
¢ You will learn how to provide answer files for the auto install process. 


What you should know... 


« You should possess basic UNIX knowledge 


- irst, you need to create your answer file (autoinstall. 
conf); below is an illustrative example, 


Choose your keyboard layout = fr 

System hostname = Box55 

Password for root account = hash password using_ 
encrypt -b 8 

Start ntpd(8) by default = y 

What timezone are you in = Indian/Reunion 
Setup a user = oadm 

Password for user = hash password. using encrypt -b- 8 

Location of sets = cd 

Set name(s) = -g* -x* +xb* 

Directory does not contain SHA256.sig. Continue without 


verification = y 


This file is hosted by a web server, here a sample: http:// 
www.mouedine.net/ autoinstall.cont. 


For more details about the answers file, you can read 
the manual for auto-install (8): man 8 auto-install. If you 
are ready, it is a good time to start on the support CD 
ROM (please, order an official CD). 

You can see in Figure 1. 


Please choose <<shel11>> to proceed. 


Next step is to enable the Dynamic Host Configuration 
Protocol (DHCP) for the connected Network Interface 


Card (NIC) [eheck installed NICs using “ifconfig” com- 
mand] as follows: 


dhclient vic0d 


Now, you can get the answer file using FTP from here: 
ftp http://www.mouedine.net/autoinstall.conf. 


Finally, just type the following command: 
install -af autoinstall.conf && eject cd0 && reboot 
WESLEY MOUEDINE ASSABY 
Wesley MOUEDINE ASSABY lives in Reunion Island, near Mauritius. 


He works at AISE-INFORMATIQUE as a network administrator. He has 
followed OpenBSD closely since 2007 (version 4.1) — it is his passion. 
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DATABASES 


Database Query 
Optimization for Huge 


Databases 


Occasionally, we have the opportunity to give the database 
engine a helping hand, and improve the performance of 

a long-running SQL query. We do this, by not performing 
the whole query in SQL. We know better than to perform 
ORDER BY in SQL (see ‘Embedded SQL.doc’); now we shall 
see how to speed up certain types of queries. 


What you will learn... 
¢ How to deal with huge databases 
- Howto speed up certain types of queries 


here are three types of query which benefit from being 
split up, in this way: 


¢ Any query which performs more than one full scan of 
a table. This inherently includes all table joins. 

¢ Any query which features more than one logical oper- 
ation in the predicate. This means any statement be- 
ginning with AND, OR and, sometimes, HAVING. 

¢ Occasionally, where the predicate features only one 
operation, but where this operation is a complex one. 


Consider the hypothetical query 


SELECT x.a, x.b, y.a, y.d 
FROM x, y 

WHERE x.a = y.a 

AND y.d > 42 


We have joined two tables, and included a numeri- 
cal comparison, as a filter for our data. The query is, of 
course, trivial, even if the tables contain millions of rows, 
but will serve as a useful template, for handling queries 
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What you should know... 


- Database optimization 


with more than one join, and more than one comparison, 
or other operation. 


At the very minimum, the query will perform a full scan 
of either x, or y, depending on how the optimizer wants 
to play it, and a full scan of the index of the other table. 
lf there is no index on the columns we've selected, mul- 
tiple scans of the second table will need to be performed, 
to find the matching data. Meanwhile, as the data is col- 
lected, each row is checked for y.d > 42. 


There are two things wrong with this scenario. Firstly, all 
of the data is read, and re-read from the disk, which is in- 
herently slow; secondly, the join, and the numerical com- 
parison, are both being performed by an SQL interpreter. 


So, what can be done, to make the query faster? 

We are assuming, as we do with all our examples, that 
we have an industrial-strength machine, with at least 2GB 
of RAM. Given this assumption, we will simplify the que- 
ry, by separating it into two subqueries, which execute as 
cursors, within our Pro*C, or embedded SQL program. 
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SELECT x.a, x.b INTO p.a, p.b 
FROM x 

SELECT y.a, y.d 

FROM y 


The first query runs to completion, giving us an array of 
two-element structures, in memory. 

Then, the second query runs, through its cursor. As it 
runs, we perform two operations, which constitute the 
original predicate. 


¢ We reject any rows where y.d is not greater than 42 

¢ We check the remaining rows, one by one, against our ar- 
ray of structures, for any row, where x.a = y.a. Those which 
match, we keep; any which don't, we reject. 


Why is this faster than allowing the database engine to 
do it? 


The answer is that for our trivial example, it probably 
isn't. However, if we’re extracting those telephone sub- 
scribers who live in a particular area, from a table contain- 
ing twenty million, and joining the result to a table of prod- 
ucts, comprising several thousand rows, the difference in 
run time can be over an order of magnitude. 


The key to performing searches on memory-based ar- 
rays Is a set of tools, the best of which we will now examine. 


Hash tables, which we consider first, are an order of 
magnitude faster than linear searches. However, they 
are not the fastest way of retrieving data from an array 
of structures. They carry an overhead of the time taken 
to actually create the table, which is approximately equal 
to the time taken to access each element. 


A similar disadvantage is shared by the binary tree, or 
B-tree, described below, which needs time to create the 
tree, whereas the binary search, or ‘divide and conquer’ 
algorithm, requires its input data to be sorted in ascend- 
ing order. 


The difference is that b-trees and binary searches are 
two orders of magnitude faster than hash tables. 


The following summary lists all four methods, applied to 
retrieving all 2 million elements from an array of 2 million 
such elements. 

The data was originally extracted from a database table, 


a single scan of which took 87 seconds. This indicates 
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that it would have taken a prohibitively long time to indi- 
vidually extract each of the 2 million rows. 


Table 1. Methods Summary 
Method 


Linear search 


Setup time Retrieval time 


Hash table 200 


Hash Tables 

Simplistically speaking, a hash table is a random access 
matrix of variables and values, where the variables are 
also the index into the matrix of values. 


Instead of saying ‘for(| = 0; | < 1000; I++)....’, and 
waiting for the sought after value to fly by, we can sim- 
ply ask the hash table for the value corresponding to 
the given variable. Unix has four hash table manipula- 
tion functions: 


hcreate(length) 
allocates space for a hash table, of size ‘length’ elements. 


hsearch(key, ENTER) 
Makes a hash table entry, for variable ‘key’ 


hsearch(key, FIND) 
Retrieves an entry described by ‘key’, from the hash table. 


hdestroy(void) 

Deletes a hash table. The data type of ‘key’ is defined by 
the typedef ENTRY, in <search.h>, which must be includ- 
ed, if we want to use the hash table functions. 


struct entry { 
char *key; 
char *data; 


a 


Since hash tables can be used with arrays of complex 
data structures, the pointers to char are an indication of 
how the hash table is implemented. 


A hash table only stores hash values, and pointers to 
the original data. This data must persist, with unchanged 


keys and addresses, throughout the life of the table. 


The data itself can change, and the pointer will still cor- 
rectly retrieve it, but its address must be constant. 
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For obvious reasons, the key must be unique, so the 
same criteria must be applied to its choice as are applied 
to choosing the primary key to a database table. If the tar- 
get array comprises the rows of a table, which has been 
extracted into memory, the primary key of the table is an 
obvious choice. 


There is one restriction on the choice of a key. 
The hash table comparison function is strcmp (), which 
means that the key has to be an ASCII string, and num- 
bers have to be represented by their ASCII values. 
However, the overhead of the extra sprintf() is negli- 
gible, compared with the saving of time, especially when 
scanning a huge array. 


By way of example, suppose that we have an array of 
1000 data structures, each of which describes a product, 
as per: 


StrUCcE Produce 4 
Char product code[10]; 
char colour([25]; 
float length; 
int style; 
float price; 
bi; 
struct product products[1000]; 


ENTRY hhash; 


Early in our code, we create the 1000 element hash ta- 
ble, like this: 


if (hcreate (1000) == 0) { 
printf (‘“Can’t allocate memory for hash table\n”); 


exit (-1); 
As we create or load the array of product structures, we 
add an extra step, to make the hash table entries: 
for(l = Of I < 1000; I++) { 
get product data(products[I]); 
hhash.key = (void *)products[I].product_ code; 
hhash.data = (void *)éproducts([1I]; 


if (nsearch(hhash, ENTER) == NULL) { 
printf (“Hash table full\n”); 


VW? 
U 


32 


J* do orher eturt.-*/ 


Some time later, we are doing some processing in an- 
other loop, and need to find the colour, corresponding to 
a product code: 


char * (0G? 


struct product temp; 


for(, = 07 7 <-2Z0000000" 4 -r) 4 
Ts 
* process a lot of customer data 
at 


pe. =- customer products [7 ].product. code; 
/* Now, we need the colour */ 


hhash.key = (void *)pc 
if((temp = hsearch(hentry, FIND)) == NULL) { 
printf(“No such entry\n”); 
} else { 
SLECpy (Customer products |i lecolour, ((stiucr 
product *)temp=>data) =>colour):. 


} 


The linear search alternative to using a hash table would 
have been an inner loop, making up to 1000 iterations, 
for every iteration of the outer loop. 


Binary Trees, or B-Trees 

The binary tree is used extensively within many data- 
bases, for the creation of indices. Creation of binary 
trees represents an extremely small overhead, and they 
are second only to the binary search in terms of data 
access time, but they provide the added advantage, 
that nodes can be deleted, and elements of the tree 
can be modified, while traversing it from a given starting 
point. The binary search, on the other hand, is just that: 
a search. 


Unix provides a set of binary tree creation and search 
routines, which have the following functionality: 


tsearcn() Adds a node to the b-tree 

cfd () Searches the b-tree for a given node 
tdelete () Deletes a node from the tree 

twalk() Traverses the tree, and performs a user- 


specified action at each 
node. 
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To support these activities, we need to supply a com- 
parison routine, identical to the type used by qsort(). 
As with gsort(), It needs to take two arguments, which 
are the nodes to compare, and needs to return —1, O or 
+1, depending on whether the nodes are equal or not. 
See “Embedded SQL.doc’” for code samples. 


Syntax for tsearch is: 
void 


*Lsearcn (void, *key, VvVold **root,.ant (*emp) (void *,; 


VOLG. *) 


The ‘key’ parameter is a pointer to the element of the 
structure, by which we will later want to search the tree. 
The rather ugly **root, is a pointer to a variable, which will 
contain the address of the root of the tree, when we have 
one. For the moment, we need to point it at an address of 
NULL, so we need the following clumsy acrobatics: 


Struct. product *xrooc? 

SLrUuCt product. ** Poor; 
xroot = NULL; /* this will get set to the root of 

the new tree */ 

‘root. = xroot; /* this must point to it */ 

Taking the earlier example, using struct product, we 

would create our b-tree, like this: 


for(i. = OF 1. < LOOOs 1+4+){ 

if (tséearch({(vord *) products [1 .product: code, {vo1d 
**) root, cmpfn) == NULL) { 

printf (“Error creating binary tree\n”); 


break; 


The syntax of the tfind() command is identical, except 
that we don’t force a NULL into the address of the root of 
the tree. 


void 


yord *))3 


‘Lh (Vvold “key, voLd **root, int (“emp) (oid *, 


Taking the same example as for the hash table, let’s as- 
sume that, some time later, we are doing some process- 
ing in another loop, and need to find the colour, corre- 
sponding to a product code: 


char * pc; 


struct product temp; 
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forty = OF 7s 200000008 44) 4 
es 
ms process a lot of customer data 
* 
pe = products[j].product code); 


/* Now, we need the colour */ 


if (tind ((void *)pe, (void **)root, cmpin). !— NULL) { 
printt ("Colour >o6é< found 12m tree\n”", 
products [1] scolour); 


} 


Note that we don’t need to perform a loop in order to find 
our colour. 


lf we need to delete a particular node, we invoke the 
tdelete() function, with the same syntax as for the pre- 
ceding commands: 
void =tind(void *key;,: void **roou,. ant “(*emp) (vod *, 


VOold *) )3 


Let ‘Il’ be the index of the array element we wish to de- 
lete, then: 


if (tdelete ( (void “) products [1] .product code; 


(void **) root, -cipin) t= NULE)4 
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printf (“Deleted >%s< from tree\n”, 
products [1] .product code) ; 


} 


Let us assume that, as part of our data manipulation, 
we need to update the data in our structures. Perhaps 
we need to decrease the price for all pink items, be- 
cause there is a weekend sale. 


To perform the equivalent of a global edit, on all the 
members of the array, individually, would take an extreme- 
ly long time if, as in real life, the array contained more than 
the trivial number of items shown above. Worse, if this ar- 
ray had to be processed for every store in the country, in- 
side another loop, the run time would be prohibitive. 


This is where we would use twalk(), which has a syn- 
tax as per: 
void twalkivo1d *root, vo1d(*action). (void *, VISIT, 


cbakoy ei 


The parameter ‘root’ points to the starting node which, in 
theory, can be any node. However, the traversal is lim- 
ited to all nodes below this root so, if we wish to visit all 
nodes, we need to set the root to the first item loaded in- 
to our tree, i.e products[O]. 


The ‘action’ parameter is a function, which we need to 
write, to tell twalk() what it should do, as it accesses each 
node. As each node is accessed, twalk() will pass to our 
function, three arguments, of types void *, VISIT, and int. 
The first argument is a pointer to the node currently being 
visited, the second is an enumerated data type, whose 
values are as follows: 


The node is visited before any of its children. 


Node is visited after its left child and before its 
right 


The node is visited after both children 


0: preorder. 


1: postorder. 


2: endorder. 


leaf. This node is a leaf 


Basically, preorder means that this is the first time the 
node has been visited, postorder means the second 
time, and endorder means the third time. 


Leaf means that this node is a leaf of the tree. 


The last argument is the level of the node, relative to the 
root of the tree, which is level zero. 
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From these parameters, our function can deduce where 
we are and what data we're looking at, and perform ap- 
propriate actions. 


lf we had defined a function modify data(), to perform 
our manipulation, then we would call twa1k() thus: 


twalk(root, modify data); 


Binary Search 

When all we want to do, is find data fast, we use the binary 
search, which is the fastest of all of the non-linear search 
techniques. It uses the familiar ‘divide and conquer’ algo- 
rithm, similar to the one used by gsort() to sort data. 


Before running bsearch(), the data must be sorted, us- 
iNg qsort (). There is only one function call to remember, 


bsearch(void “key; void “base, size. t numelmt; size t 


Slit. Sze, 1nt “(*emp) (void “> void =) 4 


As before, cmp() is the usual comparison function, and 
key is the item for which we are searching. The param- 
eter numelmt, is the length of our array, and elmt_size is 
sizeof (our data structure). It is called, quite simply, as: 


char *pc; 
SLEucr. peoducr. “pp? 

pe = products |1].product. code; 

if ((pp = beearci( (void *)pc, product, 1000, 
sizeof(struct product), cmp)) != NULL) { 

printf (“Product >%s< found in array\n”, 
PP=-product. code); 

} 


MARK SITKOWSKI 
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raster. 
Better. 
Reliable. 
Trusted by over 500 ISPs worldwide. 


Hyper is the first multimedia cache fully developed in Brazil, by Taghos. 
With Hyper, ISPs can save on network bandwidth while increasing 
content-delivery speeds, resulting in end-customer satisfaction. 


Features: 

- 24x7X365 always-on support 

- Active monitoring 

- Automatic updates 

- Appliance or license 

- Easy deployment 

- Configuration and reports via 
web interface 


Model Traffic RAM Cache S50 
T15 Up to 15 Mbps 8 GB 1x 118 : 
750 Up to 50 Mbps 8 GR 2x 118 
T100 Upto 100 Mbps 8GB 2x 116 1x 160 GB 


T150 Up to 1ISOMbps 16 GB 3x 278 1x 160 GB 
1300 Up to 300 Mbps 16 GB 5x 216 1x 240 GB 
T3500 Up to 500 Mbps 32 6B Tx 276 1x 480 GB 
T1000 Up to 1 Gbps 64 GB 10x 1 TB 1x 480 GB 


Pemote Install T2000 Up to 2 Gbps 96 GB 24x 1 TB 3x 480 GB 


Using your hardware T3000 Up to 3 Gbps 128 GB 32x 1 TB 5X 480 GB 


Visit us at WWW.taghos.com and start saving bandwidth today! 
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Inserting and 
Retrieving Data From 
Huge Databases 


The focal point of our attention, is the database, from 
which and into which we wish to transfer extremely large 


quantities of data. 


What you will learn... 
¢ How to transfer extremely large quantities of data 


hen we invite our database engine to execute a SE- 
LECT statement, two things happen: 


¢ We run our SQL statement through the database 
server’s SQL interpreter 

¢ It performs a read operation on the areas of disk con- 
taining the table or tables, which are the subject of 
our SELECT statement. 


If our choice had been a DELETE, UPDATE or INSERT 
operation, we would have been performing a disk write. 


Interpreters are not known for their lightning speed, and 
disk I/O is the slowest operation that a process can per- 
form. Therefore, we can infer that we should minimize 
the amount of work done by the SQL, and minimize the 
amount of reading or writing that the database engine 
does to the disk. Specifically: 


Don’t Do The Logic in SQL 

Consider a cursor, which is an SQL construct used for ex- 
tracting multiple rows from a database, executing the fol- 
lowing statement on a table of ten million rows: 


SELECT one, two, three FROM there 
WHERE one = 1 AND two = 2 AND three = 3 
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What you should know... 
¢ SQL Basics 


This will execute very slowly, since on every row of the 
full table scan, which we need to do, we will have to per- 
form three comparisons. It is far more efficient to trun- 
cate the predicate to perform only one comparison: 


SELECT one, two, three FROM there WHERE one = 1 


and perform the remaining two comparisons in our appli- 
cation: 


while(sqlca.sqlcode == 0) { 
EXEC SQL FETCH cur INTO :one, 
If(two != 2 || three != 3) 


stwo,; ?three; 
continue; 


Never Ask the Database Engine 

to Do ORDER BY 

ORDER BY entails the database engine doing a sort on 
your data. In order to do the sort, it has to use temporary 
disk space within its file system. If the database is not idle, 
but has other users performing operations, and the quan- 
tity of data to be sorted is huge, two things will happen: 


¢ You will run out of temporary space 
¢ The operation will take a long, long time. 
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There is a Unix utility, called gsort() which can do, ina 
few milliseconds, what any database engine can do in 
a few minutes. Further, it performs an in-place sort, so 
we don't need huge amounts of temporary space and, 
far more importantly, we not need disk I/O. 


The synopsis of gsort () IS: 


Vold qS0rt( (void. *)polnter to-date, size t num elements, 
Size. t eizecr (element); 1nt(*cmp function) (void *, 


woud *)? 


where cmp function() IS a user-supplied comparison 
function for doing the actual sorting. Although the defini- 
tion looks quite ugly, all it means is that the function has 
to conform to the following rules: 


¢ |t accepts two elements as arguments 

¢ Itreturns 0 if the elements are equal 

¢ It returns+1 if the first argument is greater than the 
second 

¢ It returns —1 if the first argument is less than the sec- 
ond. 


Now, let us assume that the cursor is executing the 
statement 


SELECT one, two, three FROM there WHERE one = 1 


but we would ideally like it to produce an ordered list, as 
produced by 


SELECT one, two, three FROM there WHERE one = 1 
ORDER BY one 


Then, instead of using the ornpER By, we would load the 
data into an array of structures, which look like 


struct data { 
int one; 
int two; 
int three; 
bi 
struct data datarray[100000]; 


Then our comparison function would look like this: 


emp data (void *pl, void *p2) f* Cmp 
data */ 
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The BSD Certification Group Inc. 
(BSDCG) is a non-profit organization 
committed to creating and 
maintaining a global certification 
standard for system administration 
on BSD based operating systems. 


BSDA: Entry-level certification suited for candidates 
with a general Unix background and at least six months of 
experience with BSD systems. 


BSDP: Advanced certification for senior system administrators 
with at least three years of experience on BSD systems. 
Successful BSDP candidates are able to demonstrate 

strong to expert skills in BSD Unix system administration. 


We're pleased to announce that after 7 months of 
negotiations and the work required to make the exam 
available in a computer based format, that the BSDA 
exam is now available at several hundred testing centers 
around the world. Paper based BSDA exams cost $75 USD, 
Computer based BSDA exams cost S150 USD, The price of 
the BSDP exams are yet to be determined. 


Payments are made through our registration website: 
hittps://register.bsdcertification.org//register/payment 


Mone information and links to our mailing lists, Linkedin 
groups, and Facebook group are available at our website: 
http://www. bsdcertification.org 


Registration for upcoming exam events i available at our 
registration website; 
hitps://register, bsdcertification.org/register/get-a-bsdeg-id 
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stcruce data: *ql,. *q23 


gl-= (struct data’ *)pl; 

G2 = (Struct data *)pz; 

if(ql->one < q2->one) return(-1); 
else if(ql->one > q2->one) return(1); 
return (0); 


} /* cmp data */ 
and gqsort() would be called like: this: 


gsort( (void ~)data p,. 100000, sizeort (struct. data)» “cmp. 


data); 
If we need to ORDER BY more than one variable, we 
need to adapt the comparison function accordingly. 


For example, if our original SQL was: 


SELECT one, two, three FROM there WHERE one = 1 
ORDER BY one, two, three 


we would need the following modification: 


ciip data(void *pl, void *p2Z) 
data */ 


/* crip 


struct data: “ol, *q2; 


gl-= (struct data’ *)pl; 

q2 = (Struct data *)p2Z; 

it (gl=>one “ G2->o0ne). return (=1)7 
else if(ql->one > q2->one) return(1); 
else if (ql->one == q2->one) { 
1f(ql->two < q2->two) return(-1); 


fa, 


else if(ql->two > q2->two) return(1); 


else if (ql->two == q2->two) { 
1f(ql->three < gq2->three) return(-1); 


else if(ql->three > q2->three) return(1); 


else if(ql->three == q2->three) { 


return (0); 


} 7* emo ‘data. */ 
lf the ‘one’ elements are equal, we make our deci- 


sion based on the ‘two’ elements and, if these latter are 
equal, we use the ‘three’ elements. 
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Run Each Cursor in a Separate Process 
It is probable that any application program for high volume 
data extraction will be using more than one cursor. If this 
is the case, then we would prefer to run all cursors simul- 
taneously, from separate processes. 

In theory, this will reduce the total run time of all of the 
cursors, to the run time of the slowest. 

In practice this depends on a number of factors: 


Does The Database Support a ‘Dirty Read’? 
lf the database locks the rows in a table, which is being 
accessed for any write operation, then our read operation 
will have to wait for the write to finish. 

Oracle will give certain concessions, if we do: 


EXEC SQL SET TRANSACTION READ ONLY; 

while Informix explicitly allows: 

EXEC SQL SET ISOLATION DIRTY READ 

and DB2 includes isolation level in the cursor definitions. 


How Many Other Applications Are Accessing 
The Same Tables? 

Obviously, if we have only one process, accessing the ar- 
ea of disk containing our tables, we will be able to achieve 
a higher read rate. The above notwithstanding, running 
multiple cursors from separate processes will always lead 
to a performance advantage. 


On a single-processor machine, the separate process- 
es will each occupy a separate slot in the process table. 
lf we have ‘n’ cursors, this will give our application ‘n’ times 
the CPU time of a single process. 


On a multi-processor machine, in addition to the above 
advantage, we will almost certainly get a processor for 
each cursor. 


Use a PREPARE Statement for Cursors 

To avoid duplicating the boilerplate code that creates a 
cursor, it is more convenient to define the SQL in a char- 
acter string, and to pass it in to one function, which makes 
a generic cursor. 


char *astring = “SELECT one, two, three FROM there \ 
WHERE one = 1 AND two = 2 AND 


three = 3”; 


char *bstring = “SELECT four, five, six FROM here 
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WHERE four != 4 AND five != 5 AND 


six l= 67s 
/* 
e We can then call the function below, like 
this: 
* SCursorSe(ascring, 1.) 
= cursors(Dst Ping, 2) 7 
my 
cursors(char *curstring, int which) fo* 


cursors */ 


Switch((pid = fork())){ 
Case S14 
printf (“Fork from cursors() (%d) failed\n”, 
When) 
perror (“Fork”) ; 
exit (-1)? 
break; 
case 0: /* This 
ig. the child process */ 
1 (Senpgrp == = 1) 
printf (“Warning: Cursor Child can’t set 
pgrp\n"”); 
} 


/* Connect to DB */ 
EXEC SQL 


CONNECT :user IDENTIFIED BY :pswd AT DB 


XYZ using :dbase; 


if(sqica.sqlcode != 0) { 
printf (“PID %d:Connection refused:%s\n”, 


getpid(), 


sgqilca.sqlerrm.sqlerrmc) ; 
exit(-1); 
} 
printf(“PID %d (%d):Connected to DB %s\n"”, 
getpid(), which, dbase); 


EXEC SQL AT DB XYZ SET TRANSACTION READ ONLY; 


/* Prepare the cursor from the incoming 


string “7 
BXEC SQL AT DB XYZ PREPARE xcur FROM 


‘curse ring: 


if(sqica.sqlcode != 0) { 
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printf (“PID %d:Can’t prepare cursor:%s\n"”, 


getpid(), 


sglca,sqilermm.sqlerimc) ; 


EXEC SQL AT DB XYZ ROLLBACK WORK RELEASE; 
exit(-1); 


/* declare the cursor */ 
EXEC SQL AT DB XYZ DECLARE gcur CURSOR FOR 
xCur; 
if(sqlica.sqlcode != 0) { 
printf (“Can’t declare cursor:%s\n”, sqlca:. 
sqlerrm.sqlerrmc) ; 
EXEC SQL AT DB XYZ ROLLBACK WORK RELEASE; 
exit(-1); 
} 
/* Now open it */ 
EXEC SQL AT DB XYZ OPEN gcur; 
if(sqlca.sqlcode != 0) { 
printf (“Can’t open cursor:%s\n”, sqlca. 
sqlerrm.sqlerrmc) ; 
EXEC SQL AT DB XYZ ROLLBACK WORK RELEASE; 
exit(-1); 
} 
/* Now, the individual cursor-specific code */ 
Switch (which) { 
case 1: 
while(! (sqlca.sqlcode == 1403 || sgqlca. 
sqlcode == 100)) { 
EXEC SQL AT DB XYZ 
FETCH gccur INTO :one, :two, :three; 
} 
break; 
case 2: 
/* feteh cursor 2. */ 
break; 
case 3: 
/*® *ecch cursor 3.4/7 
break; 


Si Cas 
default: J/* back in 
the parent process */ 


printf (“Child process %d running\n”, pid); 


break; 


/* cursors */ 
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Use the Oracle Array Cursors for Input or 
Output 

Oracle has the facility of fetching a cursor into a host vari- 
able which is an array. 


The advantages of this are considerable. With one SQL 
request, we can fetch, not just one row, but any number, 
like 20000, or 30000. Equally, we can perform an INSERT 
from an array host variable, and load 50000 rows, or how- 
ever many we feel should form our syncpoint. 


The Array FETCH is Performed Like This 
Declare all input host variables as arrays: 


#define FCH 32000 


Struct. data 
int one[FCH]; 
int two[FCH]; 
int three[FCH]; 
bi 


Define the cursor string: 


char *astring = “SELECT one, two, three FROM there \ 
WHERE one = 1 AND two = 2 AND 
three = 3”; 


¢ PREPARE, DECLARE and OPEN the cursor, as de- 
scribed above. 
¢ FETCH the cursor into the host variables. 


This is a little more complicated than a usual FETCH, since 
it is unlikely that the size of our array divides exactly into the 
number of rows in the table. We need to know how many 
rows were returned, with each iteration of the cursor. 


This is achieved by using the sqlerrd[2] member of the 
sqica structure, which holds a running total of the rows 
returned. For convenience, we define two local variables: 
int prev_fetch; /* the number of rows we 
got last time */ 

/* the number of rows in 


Int Current Tetch; 


thus Terch. */ 


We then FETCH the cursor into our array variables: 


while(! (sqlca.sqlcode == 1403 || 
EXEC 60 AT DB STG FETCH gecur INTO 


sqlca.sqlcode == 100)) { 


:data.one, 
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:data.two, 


:data.three; 


/* 32000 in one FETCH */ 
1f(sqlca.sqlcode != 0) { 
if(sgqlca.sqlcode != -1405){ /* 
ignore fetched NULL */ 
EXEC SQL AT DB XYZ CLOSE 
gccur; 
1f(sqlca.sqlcode != 0) { 
printf (“Can’t close 
cursor:%s\n”, sqlca.sqlerrm.sqlerrmc) ; 
EXEC SQL AT DB XYZ 
ROLLBACK WORK RELEASE; 
exit (—1) 7 
} 


break; 


} 
if (sqlca.sqlerrd[2] == 0) break; /* no 
data */ 
x 
* sgqlca.sqlerrd[2] is a running 
total, so we must 
* subtract the previous total, to get 
the Current 
* number fetched 
x] 
cCuprene. Tetcn = sqlcacsqlerrd (2). = 
prev. fetch; 
prev: Ttétch = sqlca.sqlerrtd(2|; 


LE(CURrent Teich == 0) bieak; 
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Interview with 


Brett Davis 


Can you please introduce yourself and explain 
when and how you got in touch with iXSystems 
and TrueNAS? 

This year marks 10 years at IX for me. | joined in late 
2003 as the first and only dedicated salesperson and 
fourth employee overall. At the time, the company had 
been around in its current iteration for a little less than a 
year, following an employee asset buyout after the Win- 
dRiver acquisition of BSDi. It was a dingy little office. 
No one was taking salaries. It was a pretty typical post 
“dot-bomb” era startup scene. What wasn't typical, 
though, were the uniqueness of the personalities and 
zealous passion for Open Source software, most specifi- 
cally, “BSD. “Making money” had been the driving force 
for any and every other for-profit company I’d worked for, 
but | immediately realized that the priorities at ixsystems 
were different. Not that they didn’t want to make money, 
but rather, money was always discussed as just a tan- 
gible side benefit of doing something they really cared 
about (furthering BSD), not a primary focus. It was an ex- 
tremely refreshing and exciting energy then, and we've 
been successful at keeping that intact some 10 years 
and many employees later. 
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TrueNAS is a good example of this, too. For those 
that may not know, TrueNAS is essentially the enter- 
prise appliance version of FreeNAS. And, for those that 
also may not know, iXsystems is the corporate spon- 
sor of the FreeNAS Project (www.freenas.org), which is 
now the world’s most popular storage OS. When taking 
over development of that project in 2010, our main ob- 
jective was to keep FreeNAS a part of the *BSD family, 
after some talk in the community about refactoring the 
code and basing it on Debian. There was never discus- 
sion of closing the source or making it paid software. 
The goal at hand was to keep it BSD and continue to 
improve it for all the world to use. This was misunder- 
stood by some at first. There was some talk of iX be- 
ing the “evil corporate overlord” with nefarious plans 
to deprive the world of this awesome, free software. 
We never let that bother us; we found it kind of funny, 
actually. Instead, we chose to speak through action, re- 
writing the software from scratch to allow the inclusion of 
ZFS and keep pace with modern FreeBSD, and we now 
have a 4+ year track record of continuing to improve it. 
And, it’s still free and always will be. 


But, throughout the process of re-writing FreeNAS, we 
learned a lot, and one of the most interesting discover- 
les was how widely FreeNAS was being used. We were 
shocked at some of the massive enterprise deployments 
we were finding out about. Although, the way we were 
finding out about them was usually pretty tragic. It would 
typically go something like this: we'd receive a desper- 
ate support request from a System Administrator at XYZ 
Fortune 500 company that had built a massive infrastruc- 
ture out of some old Dell servers, or had read an article 
about the Backblaze and was brave enough to try and 
build one for enterprise use, and had some sort of outage. 
After trying to help some of these folks, most of the time in 
vain, spending hours trying to piece together in our minds 
the hardware abomination they'd built before we could 
even then determine if we were looking at a software or 
hardware issue, we realized something: storage is hard. 
More specifically, “Software Defined Storage” is hard. 
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There are so many moving pieces and endless hardware 
combinations that if you’re building anything more than a 
simple file server, this notion of “just load this software on- 
to any old hardware you have lying around, and you can 
replace your NetApp” is really kind of a myth and a dan- 
gerous game for businesses to be playing. Software De- 
fined Networking on the other hand? That | get because 
you re basically just buying an embedded whitebox switch 
and putting software on it. The hardware is simple, and 
you can save some money. But, with storage, there are 
just too many variables. So, we realized pretty quickly 
that an enterprise appliance version of FreeNAS was nec- 
essary so that businesses could receive the kind of reli- 
ability, predictability, and enterprise-class support that is 
required for something as critical for businesses as their 
data. We could have tried to ride the whole “software de- 
fined” wave. It would have been the easy cash grab but 
would have gone against the very principles by which the 
company was founded. So, through the experience of try- 
ing to support FreeNAS, we realized it was actually the 
opposite of what businesses needed. And, that’s how the 
TrueNAS line of appliances came to be back in 2011. 


What are the main innovations of the new 
TrueNAS models compared to the previous 
generation? 

We're pretty proud of this new TrueNAS lineup, released 
in July. The first thing you'll notice is that the product line 
is slimmed down quite a bit. We've achieved this by de- 
signing a purpose-built, modular hardware platform that 
is the foundation for all three versions of the appliance. 
This design makes the appliances easier to manage 
and maintain — everything is tool-less and swappable. 
Every model is also now upgradeable to “high availabili- 
ty” by adding a second storage controller, something that 
wasn't possible with the last generation. Under the hood 
you'll find more powerful hardware across the board: 
faster CPUs with more cores, more RAM, faster con- 
trollers, better chipsets. Not to say the previous genera- 
tion was lacking in those departments, but the hardware 
is just more modern now. I’m also told it's much sexier, 
which is probably more important than any of us would 
like to admit. 


What are the main advantages of TrueNAS? 

TrueNAS’ advantages over FreeNAS are simple: True- 
NAS is a full blown appliance, specifically designed for 
enterprise storage and all its trappings. FreeNAS is just 
software. There are also some minor software differ- 
ences tied to the specific hardware we use for TrueNAS 
as well: high availability (failover), performance tuning, 
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drive failure LED notification, global fault light indication 
on the bezel (TrueNAS logo turns from white to red when 
there are system alerts), specific enclosure manage- 
ment hooks, hot spare drives, etc. Exposing these fea- 
tures to the potentially infinite number of FreeNAS builds 
just isn't possible. 


But, even more important than those differences is the 
comprehensive support available for TrueNAS. Even as 
the creators of FreeNAS, we cannot provide the qual- 
ity of support for FreeNAS that mission critical environ- 
ments require. It just isn't scalable, and were not inter- 
ested in providing something that isn’t of consistently 
high quality. We’ve watched other Software Defined 
Storage companies try, gain some success, but ultimate- 
ly fall flat once enough people catch wind of the difficult 
and painful customer experience when things go wrong. 
TrueNAS allows us to avoid that mess entirely and make 
sure our customers have the peace of mind that if some- 
thing does go South, we're always here to provide a 
prompt resolution. 


TrueNAS is also part of the VMWare Ready and Citrix 
Verified programs, meaning that it’s an officially supported 
platform for virtualized environments. This isn’t something 
we can do with the FreeNAS software, again due to the 
potential variability of the hardware. 


When compared to other traditional storage applianc- 
es, TrueNAS’ advantages are in the power of the ZFS file 
system as well as the open source development meth- 
odology. | won’t belabor the ZFS point, since that’s a 
known quantity around these parts, but the open source 
model gives us a distinct advantage over our com- 
petition, which is our accelerated development pace. 
FreeNAS has a pretty decent sized community and 
some very engaged community members. This allows 
our alpha and beta cycles to be shorter due to the con- 
stant stream of collaborative testing between our devel- 
opers and the community. No QA team could work this 
efficiently. This model blesses us with the agility to add 
new technologies and features at a much faster clip than 
our big, slow-moving competition, while giving us extra 
assurance the software is solid. TrueNAS then follows 
behind FreeNAS by a few weeks, just to allow for ad- 
ditional soak time and for our own internal QA process. 


Where do you see the product line growing in 
the near future? 

We have a lot planned for TrueNAS on the software side 
of things that are now made possible by the streamlined 
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product line, most of which are related to continuing to 
make the TrueNAS customer experience the best in the 
industry. We're looking at predictive drive failure notifica- 
tion, improvements to the support and reporting modules, 
and other tweaks that make things even easier to use 
and manage. 


Plugins are currently only available in FreeNAS be- 
cause most of them are designed for home use (PLEX, 
owncloud, sickbeard, etc), but we have plans for some 
enterprise-specific plugins on the horizon. We are also 
going to be adding the support of a few more protocols 
as well. Extending our VMWare integration by becoming 
a certified VAAI platform is also on our near-term road- 
map. These are a few things coming up that I’m at liberty 
to discuss. 


Who do you see TrueNAS competing with? 
Synology/Readynas? EMC/Netapp? 

The power of the ZFS file system automatically puts True- 
NAS (and even FreeNAS) in a class apart from the SOHO 
NAS manufacturers you mention. ZFS is among the elite 
echelon of storage file systems. If you’re using a general 
purpose storage file system like EXT3/4 or XFS for stor- 
age, or trying your luck with something as nascent and un- 
tested as ZFS on Linux, then you obviously don’t really 
care about your data. That, or you don't know any better. 


We find that we’re mostly up against the big guys you 
mention; replacing legacy installs and providing most of 
the same functionality, and similar if not better perfor- 
mance, typically with one less zero in the price tag. 


We also see some competition from the Software De- 
fined Storage players here and there, though that trend 
really seems to be on a downward arc over the last 
two years. 


Will TrueNAS ever be available as a standalone 
software license? 

We get asked this a lot, but the answer is no. We be- 
lieve that the jig is up for Software Defined Storage’s util- 
ity in enterprise IT environments now that enough people 
are discovering that the cost savings are mythical. It’s our 
opinion that offering TrueNAS as a standalone software 
to install on any commodity hardware would be a short- 
sighted move that would contrast against the best interest 
of our customers. 


FreeNAS software, however, will always remain avail- 
able for home NAS and small offices or simple, non-criti- 
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cal file servers, and/or those willing to support themselves 
with the help of the community. 


How is the VAAI (VMware primitives) 
integration going? 

From a development perspective, it's going quite well. 
| believe we'll have it unofficially available soon, but then 
we have the hard part: the VMWare certification process. 
My hand is already starting to cramp thinking of all the 
checks | have to write. 


Will iXsystems integrate flash storage (whether 
PCI-Express or SSD) into upcoming TrueNAS 
product lines? 

SSD flash storage is integrated throughout the line as 
ZIL and L2ZARC, and always has been. SSDs are also 
available for primary storage as well. We have also done 
PCl-Express flash for L2ARC in the past, however we've 
moved away from it since it is problematic for failover. 


Are there plans to provide programmatic 
management and configuration of TrueNAS 
devices, like with chef or puppet or ansible etc. 
There are, but I'll have to defer to Jordan Hubbard for de- 
tails on that one ;-) 


Summing up, please tell our Readers why 
TrueNAS is so unique and what the company 
can achieve when they decide to use it? 

TrueNAS essentially enables FreeNAS & ZFS to be uti- 
lized in a business environment with the confidence in 
knowing that the hardware is qualified and purpose-built 
for the software, tuned appropriately, and there are folks 
always waiting by the phone to assist when needed. 


It’s a powerfully-flexible and scalable line of applianc- 
es that easily integrates with heterogeneous storage en- 
vironments at nearly every tier. What that means is that 
there aren't many storage requirements that can’t be 
solved with TrueNAS. And, the ones that can’t are being 
worked on in the lab, | assure you. 


And, lastly, people can feel good knowing that an in- 
vestment in TrueNAS (or any iXsystems product for that 
matter) is an investment in the thing we're all here for, af- 
ter all: the continued development and proliferation of all 
things *BSD. 


BY LUCA FERRARI 
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While it cannot be disputed that the World 
Wide Web and the Internet has helped 

in speeding up the advancement of 
globalization in eroding national barriers — is 
there a down side to mass connectivity? 


Perry — the award winning Time journalist — entitled 

“Falling off the edge, Globalization, World Peace and 
other lies”. Therein are contained some frightening sta- 
tistics, not just concerning the inherent national angst 
and violence that arise when sudden change is imposed 
upon a country. Take India for instance, at current lev- 
els of growth it will require more than 100 years for the 
nation to reach parity with living standards in the West. 
During the Iraq war, more journalists were killed than dur- 
ing the whole of the Vietnam war. While the author makes 
clear his distaste for the globalization agenda from the 
viewpoint of a hardened war correspondent, the parallels 
between the socio-economic-political and technological 
universes could not be less contrasting. 

While the crux of the anti-globalization argument rests 
on inequality — the “haves” versus the “have not’s’, the re- 
source rich versus the resource poor, the strong against 
the weak — technology has almost always been consid- 
ered the great leveller. And with the similar ethical passion 
as the politician, banker or plutocrat, the argument has 
always been that wealth trickles down — be that financial 
wealth or educational knowledge and skill. It is however 
becoming more clear in these recent years of austerity 
that this model is broken — as the gap between rich and 
poor is widening not decreasing. And so it is with technol- 
ogy. Apart from the few who really want to get their hands 
dirty, the majority of people are happy with their Face- 
book, Twitter, Email and website access until of course 
when the communications go down or a virus attacks in 
which case all hell breaks loose. Our Internet connection 
here has been very unreliable over the past few months. 
Mainly | suspect this is happening due to the roll-out of Fi- 
ber in the area — at one point | was experiencing the light- 
ening fast connection speed of 100Kbps and all without 
the comforting sound of a pair of modems handshaking. 
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My teenage daughter was climbing the walls, and life was 
definitely more — not less stressful. 

Now, | could be accused of over dramatization here — 
after all, wars, rebellions and uprisings kill people — a life- 
less Ipad or PC will rarely be cited as the cause of death 
on a coroner's certificate. But dig beneath the surface, 
and the same subtle dynamic is at play — you must be 
part of the crowd or else you are an outsider. Join the 
technological arms race or be flattened by the opposition. 
Sup at the communal bowl of the Internet of Things (loTs). 
And then you may pick up some nasty diseases while you 
are at it. The problem is not the connectivity; it is the un- 
charted territory that goes with it. What is sauce for the 
goose is sauce for the gander and the old establishment 
is reeling as the openness of new media is exposing their 
weaknesses quicker than the traditional channels ever 
could. Politicians and legislators are struggling to keep 
hold as the criminal and terrorist move from the more vis- 
ceral bank robberies and bombings to fraud and electron- 
ic infrastructure attacks. The definition of property is being 
redefined by mega-corporations, and personal data be- 
came public knowledge years ago. And here lies the rub — 
while not explicitly stated by Alex Parry, you can almost 
touch the intellectual exasperation on every page of his 
book — where has the ethical order gone? Why are we de- 
scending into chaos? 

lt would be easy then, to take the simplistic view and say 
the Internet is broken and try and regain it. This seems to 
be the current philosophy amongst the powers that are ex- 
isting and thankfully we are not yet at the stage where we 
have UN peace-keepers patrolling the World Wide Web. 
However, the insidious creep of government vampiring our 
network traffic and communications is continuously un- 
abated at the same time that Internet censorship is on the 
rise. As | have stated many times, politicians and lawyers 
are best kept as far away from regulating technology as 
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possible as they rarely appreciate the subtleties of robust 
engineering or human ingenuity. What is more troubling is 
that we can expect the same global solutions to the global 
problems as we have had over the past hundred years 
— in other words sweet nothing other than to make the 
problem much worse in the long term. The technological 
age was meant to bring in more free time, a paperless of- 
fice, free electricity and a better quality of life for everyone. 
For a few, this is the case, but the majority of the world 
remains poor, hungry and dispossessed. We can send a 
man to the moon but can’t manage to get clean drinking 
water to the 780 million people who need it. And where 
do the global leaders house their outsourcing operations? 
Where it is cheap — like India of course. 

In the late 80’s and early 90's, there was a great trend 
for multinationals to be ethically led. | had the pleasure 
of working with two visionaries — Michael Kidron and 
Bela Hatvany — in a small tech start-up that monitored 
the ethical behavior of multinationals and blue chips. 
Sadly, it was ahead of its time and didn’t make the grade 
in the harsh world of pre-internet electronic publishing. 
Sites like Wikileaks today have global reach, but the pace 
of change is too slow and in the arms race a small orga- 
nization has no traction with a huge PR machine — espe- 
cially if it has access to other forms of media or worse to 
be the ear of a government. That is why it is so critical 
that the Internet remains a pure voice for all. This curse of 
“managing expectations” has penetrated through corpo- 
rate website forums, blogs and message feeds to the ex- 
tent that PR companies are now advertising on commer- 
cial radio offering to manage your reputation on the web. 
Software tools are used to monitor Twitter and Facebook 
in the guise of customer service, but you can bet that the 
real motive behind this is to silence any real criticism. Na- 
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ture abhors a vacuum, and as we have allowed commer- 
cial interests to dominate the infrastructure of the Internet, 
now so shall they dominate the ethical landscape as well? 
And as it is with the global erosion of sovereignty, so will it 
be with our meta-data? The majority will go along with the 
agenda, not realizing that the first call of any reputable re- 
cruiter or law enforcement officer will get their Facebook 
pages. It is very easy to click in haste and repent at leisure. 
And far away from the suggestion that we need an Inter- 
net policeman, we do however need stronger ethics and 
vision for the Internet. Not just on the corporate level, but 
for the 70% who are not online and cannot speak for them- 
selves. From where | stand, that is the only positive point 
| can see — a bunch of connected, networked individuals 
who make a stand and say with one voice “This is not right”. 
If the Internet looses its edginess, and becomes just an- 
other media-barons paradise on a global scale, we really 
will have lost any voice as the sticky ocean of mediocrity, 
alternative agenda, doublespeak and compliance smoth- 
ers any dissenting opinion. Mike Kidron and Bela Hatvany 
had it right 30 years ago — the deeds of large corporates, 
governments and institutions need to be truly independent- 
monitoring — and they need to be held to account. 


ROB SOMERVILLE 

Rob Somerville has been passionate about technology since his early 
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worked in many corporate sectors including finance, automotive, air- 
lines, government and media in a variety of roles from technical sup- 
port, system administrator, developer, systems integrator and IT man- 
ager. He has moved on from CP/M and nixie tubes but keeps a solder- 
ing iron handy just in case. 
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Antonio Francesco 


Gentile 


Please introduce yourself to our readers. 

| am a software and network engineer. Now I’m working 
with the National Center of Research (ICAR) area net- 
working in Cosenza as the network manager. | graduated 
from the University of Calabria, specifically at the DIMES 
with the “Culture Lab” (http://culture.deis.unical.it) in the 
Department of Telematics. Currently, I’m collaborating 
with this department on several projects related to mesh 
networks, and with computer science associations “Hack- 
lab Cosenza” (http://hacklab.cosenzainrete.it/) and “Verde 
Binario” (http://www.verdebinario.org/). Finally, I’m a free- 
lance columnist for Italian magazines “Linux&C” (http:// 
www.oltrelinux.com/) and “Linux Magazine” (http://www. 
linux-magazine.it/). | am very interested in the develop- 
ments in mesh technology, security in wireless networks 
and the integration of complex services in MANs. | also 
work on systems with a focus on RTP and Asterisk VOIP 
made services. Now | live in Calabria, in the deep south 
of Italy, but | have traveled throughout my country for work 
for five years. Now I’m back in my hometown. I'll see what 
the future holds for me... 


Could you tell us more about your background? 
My first experience with a computer was an Atari XE Sys- 
tem for a course in BASIC programming | took in middle 
school. It was the era of the legendary Commodore-64. 


In high school, | worked purely on Windows machines 
(then 3.1 and 3.11 and MS-DOS). In 1997, | had my 
first encounter with Unix systems, with RED HAT 1.0, 
the legendary BAT. | had many difficulties installing with 
dual boot :). 


In college, | continued to work more frequently with 
Linux systems until in 2003, | worked with FreeBSD for 
the first time. Since then, being directed by my studies to 
the security and performance of networks (wireless and 
not), | have almost completely abandoned the systems of 
the house of Redmond. 
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| have used various programming languages, such as 
PHP, C, C++, Java, Perl, Python, but today if it’s possible, 
| always use shell scripting and Python. 


Please tell us about your proudest 
achievements. 

I'm happy to be involved in various activities related to the 
spread of Open Source software. First of all, | work as a 
teaching assistant in the courses of Telematics at the Uni- 
versity of Calabria, where | like to make simple real archi- 
tecture, so as to see that all the theory being studied is 
used to implement concrete infrastructure. 


| like to work with different magazines because it allows 
me to extend the range of people that | can reach, as if | 
were in a distance course. 


From the business point of view, however, I’m pleased to 
have completed a 20-domain migration of email services, 
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DNS, and web sites with hundreds of users over the 
course of one week for a customer. Finally for another 
company, | created a series of useful networks to con- 
nect thirty locations throughout the country. Obviously, 
| am very happy at my current job as a network adminis- 
trator at the CNR. 


Please tell our readers, what does the future of 

*BSD OSes look like? 

From the first encounter, I’ve always admired a lot of the 
BSD operating systems. OpenBSD is a constant refer- 
ence for anyone involved in security. | use it to imple- 
ment my firewall, as well as Linux with iptables. FreeB- 
SD is a milestone to achieve a robust and secure server, 
much easier to manage than older releases. NetBSD can 
now compete with Debian as far as the number of plat- 
forms supported via installation. So much for the ‘main’ 
distributions. Distributions such as DesktopBSD and PC- 
BSD are geared to novice users, with superb results in 
my opinion, and in addition to the stability of BSD have 
simple and friendly interfaces. MOnowall and pfSense are 
ideal for building robust and reliable firewall systems, but 
at the same time easy to handle, and finally FreeNAS is 
an excellent alternative to proprietary storage systems. 
From the above, | believe that BSD systems will only grow 
over time and earn more and more consistent slices of the 
market, both among individual users and big companies. 


From your point of view, what are the best 
capabilities of *BSD? 

Certainly not to underestimate the fine documentation al- 
ways available online, which represents almost a pecu- 
liarity of BSD systems, some of which were among the 
first to get on the field a full, well-organized documen- 
tation. The system is “reasoned” and well built, with a 
clear division of areas “root space” and “user space”. It 
should also be taken into great consideration the excel- 
lent system software installation, split between packages 
and ports, something that no other Unix distro can boast. 
Finally, it's necessary to recognize the reliability of BSD 
systems. Subject to proper management of security 
patches and updates, there are servers that have held up 
for years without significant management intervention. 


What was/is your best tool to work with? 

In truth, | use several tools in my daily life, such as the 
Nano text editor (although | often use Kate on KDE, my 
favorite desktop environment), GNS3 to simulate com- 
plex networks and test the operation before moving to the 
production environment. Those are my regular working 
tools. However, | also use virtualizers like VirtualBox and 
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KVM, as well as VMware. | tend to virtualize my servers if 
there are no particular contraindications, in order to man- 
age crashes and backups better, for example. If | have to 
teach or write technical or research papers, then Emacs 
and LaTeX become my allies. 


What is the best advice for those who want to 
use *BSD OS and why should they? 

When you use a BSD-like system, it is necessary to be- 
come familiar and have a greater awareness of the sys- 
tem as users become experts and not “mere users” of per- 
sonal computers. 


BSD systems, as well as those like Linux, force us- 
ers to make this initial effort, but it pays off over time 
with a deeper knowledge of the subject and the ability to 
manage systems robustly and reliably. Thanks to this, in 
the event you become professionals, you can manage 
high-level architectures. If you add to this the fact that 
the systems are open source and have splendid docu- 
mentation, it is just time to start the download for the first 
installation :). 


What are you looking for in terms of career 
development? 

| would love to work in between the corporate world and 
the university, collaborating on the development of inno- 
vative projects while simultaneously teaching and writing 
publications in trade magazines. 


It's complex and very challenging, | admit, but it is im- 
portant to believe in what we do and what we like to do. 
It's advice that, in my small way, | can give to all readers. 


How do you want to improve yourself in the 
next year? 

| think | will continue my normal life, really, maybe con- 
solidating my working relationships at the CNR. | like to 
look up a lot of docs online and study a lot of books if 
the argument proves to be interesting. In particular, I’m 
working on Mesh systems, and | believe that this tech- 
nology has great potential. One very obvious use is to 
provide Internet connections in locations not covered by 
the national network. It can also be used to interconnect 
networks of sensors physically far apart for very respon- 
sive monitoring. 
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of the talks | attended.” 
— David Hollis, Consultant, Re 
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put you on the fast track.” 
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—Rahul Gupte, Associate Direct 
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www.BigDataTechCon.com 


Come to Big Data TechCon to learn the best ways to: 


Process real-time data pouring into 
your organization 


Master Big Data tools and technologies like Hadoop, 
Map/Reduce, hbase, Cassandra, NoSQL databases 
and more! 


Learn how to integrate data collection technologies 
with data analytics and predictive analysis tools to 
produce the kind of workable information and re- 
ports your organization needs! 


A BZ Media Event # @ i & Big Data TechCon 


Big Data TechCon™ is a trademark of BZ Media LLC. 


Collect, sort and store massive quantities of 
structured and unstructured data. 


Looking for Hadoop training? We have several 
Hadoop tutorials and dozens of Hadoop classes to 
get you started — or advanced classes to take you to 
the next level! 


Understand HOW to leverage Big Data to help your 
organization today 


Become a Big Data Master! 


